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Project  Goal  Objectives  N  A  V ^  I  R 


•  Overall  Goal:  understand  the  unique  cognitive  mechanisms  that  should  be  employed  to 
optimize  collaborative  decision-making  activity  in  a  geographically  distributed  and 
time-delayed  situation 

•  Objectives:  (1)  to  understand  the  cognitive  process  of  building  knowledge  in  an  asynchronous, 

distributed  collaboration  environment 

(2)  to  develop  an  empirically-based  theory  of  collaboration,  including  knowledge 
building  components,  during  asynchronous,  distributed  collaboration 

(3)  to  understand  how  agents  can  support  humans  in  achieving  collaborative 
knowledge  during  asynchronous,  distributed  collaborative  problem  solving 

•  FY  03  Objectives: 

(1)  Conduct  experiments  in  collaborative  problem  solving  in  both  face-to-face  and 
asynchronous,  distributed  environments  to  understand  the  unique  cognitive 
processes  within  asynchronous,  distributed  collaboration 

(2)  Update  preliminary  conceptual  model  of  collaboration  based  on  experimental 

results  0 


Definition  of  Key  Terms 
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•  Collaboration  =  the  process  of  shared  creation:  two  or  more  individuals  with  complementary 
skills  interacting  to  create  a  shared  understanding  that  none  had  previously  possessed  or  could 
have  come  to  on  their  own  (Schrage,  1990) 

-  The  cognitive  aspects  of  joint  problem  solving  for  the  purpose  of  attaining 
knowledge  sufficient  to  complete  the  common  task. 

•  Asynchronous  Distributed  Collaboration  =  a  cohesive  group  of  individuals  working  at 
different  times  and  at  different  locations  to  solve  a  common  task  (Baecker,  1993) 

•  Knowledge  Building  =  process  through  which  we  increase  both  our  individual  and  our 
common  understanding  (Wells,  1999)  ;  theory  of  learning,  which  emphasizes  the 
collaborative  construction  of  knowledge  by  a  group  of  learners  (McLean,  1999) 

•  Collaborative  Knowledge  =  group  negotiation  of  different  perspectives  resulting  in  a 
broader  and  deeper  understanding  (Stahl,  2000;  Warner  &  Vanderwalker,  2002) 
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Definition  of  Key  Terms 
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•  Shared  Understanding  =  agreement  by  the  group  achieved  through  clarification  of  differences 
in  interpretation  and  terminology  (Stahl,  2000) 

•  Mental  Model  =  a  knowledge  structure  that  represents  information  (Norman,  1983) 

•  Shared  Mental  Models  =  organized  knowledge  members  have  in  common  regarding  the 
task  (Cannon-Bowers,  et  al.,  1993) 

•Agent  =  software  that  carries  out  some  set  of  operations  on  behalf  of  a  user 
or  another  program  with  some  degree  of  independence  or  autonomy,  and  in  so  doing, 
employ  some  knowledge  or  representation  of  the  user’s  goals  or  desires  (Franklin  & 

Graesser,  1996) 
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Expected  Final  Products 
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•  Empirical  Data  /  Journal  Articles 

•  Describing  the  cognitive  processes  of  building  knowledge  with  distributed 
team  members  engaged  in  asynchronous,  quick  response  collaboration 

•  The  data,  derived  from  Phase  I,  and  II  experiments,  will  be  documented 
in  journal  articles 

•  Empirically-based  Theory  For  Asynchronous,  Distributed  Collaboration  including 

Knowledge  Components 

•  Theory  will  be  documented  in  a  journal  article 

•  Empirical  Data  On  Agent  Support  For  Achieying  Collaboratiye  Knowledge  In 

Asynchronous,  Distributed  Collaboration 

•  The  agent  support  experiments  (Phase  III)  will  be  documented  in  a  journal  article 
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Conceptual  or  Computational 
Models  Developed/Planned 
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Preliminary  Conceptual  Model  of  Collaboration 


Level  1 
(Stages) 


Team  Orientatim 

getting  acquainted 
clarifying  task 
initial  attitudes 


Model 


Level  2 
(Processes) 


Individual 

Understanding 


I  I  =  Knowledge 
Building 


Articulatq  ):o  Team 

Accumulation 
of  facts  (Team) 


Select 


&  Discii 

A 


Update 


Team  Conflict 

decision  alternatives 
criticism  of  alternatives. 


of 


Decision  Alternatives 
Criticism  of  Alternatives 
(with  rationale) 


Iteration  loop  for  selecting  decisi 


Emergence  of 
Team  Decisions 

•  decisions  emerge 
from  team^ 


Negotiate  Perspectives 


alternatives 


Collaborative 

Knowledge 


Establish 


Team  Shared 
Understanding 


on  I  alternatives 


Reinforcemenl 


of  Team  Decisions! 


team  consensus 


Implement 
Decision  To  Solve 
Task 
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Research  Questions 
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•  What  is  the  knowledge  building  process  humans  use  in  asynchronous,  distributed 

collaborative  problem  solving? 

•  Understand  the  effect  of  the  following  variables  on  the  knowledge  building  process 

during  asynchronous,  distributed  collaboration? 

*  Collaboration  Modes  (  face-to-face  versus  asynchronous,  distributed) 

*  Knowledge  Distribution  (homogeneous  versus  heterogeneous) 

*  Problem  Solving  Domains  (static  knowledge  versus  dynamic  knowledge) 

•  What  forms  of  agent  support  facilitate  knowledge  building  in  asynchronous, 

distributed  collaborative  problem  solving? 
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Experiments 
(Planned  /  Completed) 
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Overview  of  Phase  I,  If  and  III  Experiments 

•  Phase  I  Objective  =  collect  empirical  data  on  the  knowledge  building  process  in  a 
collaborative  problem  solving  domain  (group  survival  scenario)  during  asynchronous, 
distributed  collaboration 

-  Examine  the  effects  of  collaboration  mode(face-to-face  vs  asynchronous,  distributed), 
and  knowledge  distribution  (homogeneous  vs  heterogeneous) 

•  Phase  II  Objective  =  building  on  phase  I  examine  the  effects  of  different  problem  solving 
domains  (static  vs  dynamic  knowledge)  on  the  knowledge  building  process 

•  Phase  III  Objective  =  within  the  context  of  the  revised  theory  for  knowledge  building 
in  asynchronous,  distributed  collaboration  (based  on  Phase  I  and  II  results),  define  and 
empirically  examine  the  beneficial  forms  of  agent  support? 
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Team  Problem  Solving  Task  |s^  A  I  R 

Survival  Scenario 


Your  group  has  secured  the  use  of  the  company  jet  to  fly  over  the  Rocky  Mountains  for  a  business 
meeting  on  the  other  side.  While  flying  over  an  uninhabited  stretch  of  mountains  a  problem  with  an 
engine  and  rough  winds  force  the  pilot  to  fly  low,  causing  loss  of  radio  contact.  Some  time  later  a 
severe  gust  of  wind  causes  the  plane  to  veer  into  the  top  of  a  tree.  The  contact  with  the  tree  rips  part 
of  the  underside  of  the  wing  causing  the  pilot  to  lose  control  and  the  plane  to  veer  toward  the  side  of 
the  mountain.  After  your  small  light  aircraft  crashes  your  group,  wearing  business  clothing,  is 
stranded  on  a  forested  mountain  in  appalling  winter  weather  (snow  covered,  sub-freezing 
conditions)  at  least  200  miles  from  civilization  (you  are  not  sure  of  your  whereabouts  and  radio 
contact  was  lost  one  hour  before  you  crashed,  so  the  search  operation  has  no  precise  idea  of  your 
location).  The  plane  is  about  to  burst  into  flames  and  you  have  a  few  moments  to  gather  some  items. 
Aside  from  the  clothes  you  are  wearing  which  do  not  include  coats,  you  have  no  other  items.  It  is 
possible  that  you  may  be  within  mobile  phone  signal  range  but  unlikely. 

Your  aim  is  to  survive  as  a  group  until  rescued.  From  the  following  list  choose  just  ten  items  that  you 
would  take  from  the  plane,  after  which  it  and  everything  inside  will  be  destroyed  by  fire. 

First  take  10  minutes  by  yourself  and  come  up  with  your  own  list  of  ten  items. 

Then  as  a  group  discuss  and  agree  on  a  10  item  list  on  behalf  of  the  group  (unlimited  time). 
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SURVIVAL  SCENARIO  Item  List 
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Choose  ten  from  the  following  list  of  items  -  splitting  or  only  taking  part  of  items  is  not  permitted: 


•Pack  of  6  boxes  x  50  matches, 

•Roll  of  polythene  sheeting  3yrd  x  2yrd 
•1  case  of  beer  (24  12  oz  bottles  in  total) 

•1  bottle  of  brandy 

•1  12  pack  of  bottled  spring  water  (twelve  liters  in  total) 

•Small  toolbox  containing  hammer,  screwdriver  set, 
adjustable  wrench,  hacksaw  and  large  penknife, 

•Box  of  distress  signal  flares, 

•Small  basic  first-aid  kit  containing  plasters,  bandages, 
antiseptic  ointment,  small  pair  of  scissors  and 
painkiller  tablets, 

•Mobile  phone  with  battery  half-charged, 

•Clockwork  transistor  radio  (receive  only), 

•Gallon  container  full  of  fresh  water, 

•Box  of  36  -  1,8  oz  chocolate  bars, 

•Shovel, 

•Short  hand-held  axe, 

•Handgun  with  magazine  of  20  rounds, 

•20yrd  of  4501b  nylon  rope, 

•Box  of  24  -  loz  bags  of  peanuts, 

•Bag  of  10  mixed  daily  newspapers, 

•Box  of  tissues, 

•Bag  of  20  fresh  apples, 

•Electronic  calculator, 

•Laptop  computer  with  infrared  port,  modem,  unknown  software 
and  data,  and  unknown  battery  life, 

•Inflatable  4-person  life  raft  (yellow  colored,  with  paddles), 
•Compass, 


•Large  full  Aerosol  can  of  insect  killer  spray, 

•Small  half-full  aerosol  can  of  air  freshener  spray, 
•Notebook  and  pencil, 

•Box  of  size  8  women’s  promotional  pink  ’Barbie’  branded 
fleece-lined  track  suits  (2  suits  total), 

•Gift  hamper  containing  half-bottle  champagne,  large  tin  of 
luxury  biscuits,  box  of  6  -  3  oz  mince  pies,  1  oz  tin  of  caviar 
without  a  ring-pull,  a  12  oz  tin  of  ham  without  a  ring-pull, 
and  a  18  oz  Christmas  pudding, 

•Traveling  games  compendium  containing  chess, 
backgammon  and  draughts, 

•Sewing  kit, 

•Whistle, 

•Flashlight  with  a  set  of  spare  batteries, 

•Box  of  50  night-light  6hr  candles, 

•Bag  of  6  large  blankets, 

•2  rolls  of  duct  tape, 

•1  carton  of  cigarettes, 

•  8x32  waterproof  binoculars, 
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Experiments 
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Phase  I  Experiments 

Independent  variables: 

-  Collaboration  Mode  (face-to-face  vs  asynchronous,  distributed) 


*  Face-to-Face  =  team  interacts  synchronously  with  each  other  through  speech 

*  Asynchronous,  Distributed  =  team  interacts  with  each  other  at  different  times 

and  from  different  locations  through  a  text  based  web  forum 

-  Knowledse  Distribution  (homogeneous  vs  heterogeneous) 

*  Homogeneous  =  the  members  of  the  team  have  all  survival  knowledge  in  common 

*  Heterogeneous  =  the  members  of  the  team  all  have  both  some  survival  knowledge  in 

common  and  some  uniquely  held  survival  knowledge 

Phase  II  Experiments 

Independent  variables: 

-  Collaboration  Mode  &  Knowledge  Distribution 

-  Domain  (static  knowledge  vs  dynamic  knowledge) 

*  static  knowledge  =  all  38  survival  items  remain  the  same  during  the  collaborative  problem 

solving  task. 

*  dynamic  knowledge  =  some  of  the  38  survival  items  change  during  the  collaborative 

problem  solving  task. 


EXPERIMENTAL  DESIGN  EOR 
PHASE  I  AND  II  EXPERIMENTS 

(  2x2x2  randomized  factorial) 
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Knowledge  Distribution 

Asynchronous, 

Collaboration  Distributed 
Mode  (text) 

Face-to-Face 

(speech) 


Static 


Domain  _ 

Dynamic 


Homogeneous 

Heterogeneous 

Homogeneous 

Heterogeneous 

Gp  1 

Gp  17 

Gp  33 

Gp49 

* 

* 

* 

* 

* 

* 

* 

* 

Gp8 

Gp24 

Gp  40 

Gp  56 

Gp9 

Gp  25 

Gp41 

Gp  57 

* 

* 

* 

* 

* 

* 

* 

* 

Gp  16 

Gp  32 

Gp48 

Gp  64 

DEPENDENT  VARIABLES: 


Phase  I 

•32  groups  total 

•  3  subjects  /  group 

•  96  subjects  total 

Phase  II 
•32  groups  total 

•  3  subjects  /  group 

•  96  subjects  total 


-  Forum  text  and  face-to-face  audio  /  video  recordings  including  time  stamp  per  response 
(I.e.  text  and  speech) 

-  Thinking  Aloud  Protocol  (concurrent  verbalizations)  for  asynchronous,  distributed  teams 

-  Total  time  to  successfully  complete  the  problem-solving  task  (time  from  the  beginning  of 
the  task  until  task  completion) 

-  Collaboration  Maps  (pre  /  post  session  -  subjects  construct  a  map  of  their  view  of  the 
stages  &  processes  of  team  collaboration) 


-  Subjective  Questionnaire  -  measuring  expertise,  trust  between  team  members,  and  general 
collaboration  opinions  among  members 


HYPOTHESES 
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•  Phase  I  Experiment 

Ho:  no  significant  difference  between  face-to-face  and  asynchronous,  distributed 

collaboration  modes  on  the  knowledge  building  process  in  a  collaborative  problem 
solving  domain 

Ho:  no  significant  difference  between  homogeneous  and  heterogeneous  knowledge 

distribution  on  the  knowledge  building  process  in  a  collaborative  problem  solving 
domain 

Ho:  no  significant  interaction  between  collaboration  modes  and  knowledge  distribution  on 
the  knowledge  building  process 

Ho:  no  significant  difference  in  time  or  frequency  within  each  knowledge  state  across 
collaboration  mode  and  knowledge  distribution  conditions 

Ho:  no  convergence  of  individual  mental  models  with  regard  to  collaboration  stages  and 
knowledge  processes 

•  Phase  II  Experiment 

Ho:  no  significant  difference  between  static  and  dynamic  knowledge  on  the  knowledge  building 
process  in  a  collaborative  problem  solving  domain 
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•  Member  National  &  International 
Research  Panels 

•  Recent  efforts:  CASC,  Agent 
Learning,  ADSS,  ANGEL,  SCC 

•Tools:  local  web  server.  Pathfinder, 
Agent  development  toolsets,  Statistica 
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•  Joint  University  efforts  (e,g,  Penn  State) 


Collaboration  Maps  (example) 


Experience  &  Capabilities 

^  in*  Over  25  Years  Experience  in 

-[-7  Decision  Making  /  Automation  Research 


^  j]  2  ^  a#  •  J 


ilowrwr  tit— llVy^D 
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COGNITIVE  AND  AUTOMATION 
RESEARCH  LAB  (CARL) 


Asynchronous^  Distributed 
Collaboration  Stations 


Face-to-Face 
Collaboration  Area 


Team  Understand , 

-  -  -  -Problem  Definition 

Problem  _Team  Shared 

^Develop 

Understanding 

Individnal  Decision  ^ - 

'Accnmnlation  of  Facts  by 

Alternatives  &  Rationale 

Team 

T  Discnss 

Welsh  Ontions _ k. 

Negotiate  Perspectives  of  alternatives 

Team  Consensns _ Ustablish - Team  Shared 

Implement 

Understanding 

Te^  Decision  To  Solve 

Task 

Web  Site  Knowledge  for 

Knowledge  Distribution  Conditions  N  A  V A  I  R 

(Homogeneous  /  Heterogeneous) 


First  Aid 


-  Sprains  and  Fractures 

-  Bleeding 

-  Hypothermia 

-  Snow  blindness 


Food 


-  Shoek 

-  Dehydration 

-  Psyehologieal  well-being 


Table  1  contains  nutritional  information  about 
various  types  of  food  and  alcoholic  beverages 

-  Beverages 
Water 
Alcohol 


Rescue  and  Escape  * 

Rescue 

Visual  Signals  Examples 

-  Fire 

-  Smoke 

-  Flare  gun  and  flares 

-  Mirrors  or  Shiny  Objeets 

-  Flashlight 

Audio  Signals  Examples 
-Whistles  -  Mobile  Phone 
-  Gunshots  -  Banging  objeets 


-Nature's  Food  -  getting  food  from  the 
enviroment 

Hunting  animals  ice  fishing 
Trapping  animals  Fruits,  plants  and  nuts 


Escape 

-Compass 

-  Maps 
-Sled 

-  Snowshoes 

-  Visibility 


Shelter 

Shelter  types  (examples): 

-  Fallen  Tree  Shelter 

-  Fallen  Tree  Shelter 

-  Tree  Pit  Shelter 

Warmth 

-  Fire  Basies 

-  Starting  a  Fire 

-  Helpful  Fire  Starting  Tips 

-  Insulating  the  Shelter 
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*  Common  knowledge  in  heterogeneous  teams 


Team  Collaboration  Map 

Template 

(Pre  &  Post  Session) 
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Collaboration  Stages 


•Implement  Team 
Decision  To  Solve 
Task 

•Weigh  Options 

•  Team 
Orientation 

•Team  Consensus 

•Team 

Leader 


•Team  Decisions 

•Team  Understand 
Problem 

•Individual  Decision 
Alternatives  & 
Rationale 

•Additional 
stages  (you  label) 


Types  of  Knowledge 

•  Problem  Definition 


•Domain 

•Collaborative 

Experience 

Knowledge  (I.e,  deeper 
team  understanding  of 

•Team  Shared 

decisions) 

Understanding 

•Accumulation  of 

•Life 

Facts  by  Team 

Experience 

•Individual 

•Critique  of 

Understanding 

alternatives 

•Additional 

&  rationale 

types  (you  label) 

Process  to  Achieve 
Achieve  •  additional 
Update  (you  label) 
Read  •  Select 

Iterate  •  Develop 

Discuss  •  Establish 


•Talk 

•Negotiate  Perspectives 
.Qf 

Connecting  Arrows 


Team  Collaboration  Map  Example  (post  session) 


•Team  Understand 
Problem 


Develop 


•  Problem  Definition 

▲ 

•  Develop 

•Team  Shared 
Understanding 

▲ 

•Talk 

▼ 

•Accumulation  of 
Facts  by  Team 


Iterate 
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snectives 


i 


•Individual  Decision  _ ^  •Weigh  Options 

Alternatives  &  •Discuss 
Rationale 


•Individual 
Understanding 

•Domain 
Experience 


Talk 


•Life 
Experience 


Thought 


•Problem 
Solving  Skills 

•Team  Shared 
Understanding 

•Critique  of 
alternatives 
&  rationale 


•Negotiate  Perspectives 
of  alternatives 

- ►  •Team  Consensus 

•Group  Dynamic 


Achieve 


Establish 


•Implement  Team 
Decision  To  Solve 
Task 


Collaboration  Stages 


Types  of  Knowledge 


•Implement  Team 
Decision  To  Solve 
Task 

•Weigh  Options 

•  Team 
Orientation 

•Team  Consensus 

•Team 

Leader 


•Team  Decisions 

•Team  Understand 
Problem 

•Individual  Decision 
Alternatives  & 
Rationale 

•Additional 
stages  (you  label) 


•Domain 

Experience 

•Team  Shared 
Understanding 

•Life 

Experience 

•Critique  of 
alternatives 
&  rationale 


•  Problem  Definition 

•Collaborative 
Knowledge  (I.e.  deeper 
team  understanding  of 
decisions) 

•Aeeufflulatlon  of 
Facts  by  Team 


•Individual 
Understanding 
•Additional 
types  (you  label) 


Process  to  Achieve 

•  Achieve 

•  additional 

•  Update 

(you  label) 

•  Read 

•  Select 

•  Iterate 

•  Develop 

•Discuss 

•  Establish 

•  Talk 

1  •Negotiate  Perspectives 
L. of  alternatives _ 


Connecting  Arrows 

- ►  f 
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TYPES  OF  DATA  ANALYSES  N  AV I  R 

Phase  I,  II  and  III  Experiments 


•  Communication  Analyses  with  Pathfinder  tool  -  identiHcation  and  representation  of  knowledge 
state  types,  and  knowledge  building  transformation  processes  compared  across  collaboration 
mode  and  knowledge  distribution.  Compare  results  to  preliminary  conceptual  model. 


•  Verbal  protocol  analyses  with  Pathfinder  tool  -  additional  information  on  knowledge  states  and 
transformation  processes  with  asynchronous,  distributed  teams.  Compare  to  conceptual  model. 


•  Parametric  statistics  for  analyzing  time,  and  frequency  within  each  knowledge  state  across 
collaboration  mode  and  knowledge  distribution  conditions  in  addition  to  total  time  to  complete 
task  and  questionnaire  data. 


•  Collaboration  Maps  -  determine  the  degree  of  convergence  between  individual  mental  model ‘s 
regarding  collaboration  stages  and  processes.  In  addition,  compare  how  an  individual  thinks  a 
group  makes  a  decision  in  a  collaborative  setting  and  how  the  group  actually  performs. 
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Percent  Frequency  By  Collaboration  Stages 


□  F2F  Homogeneous  □  AsychDistr  Heter. 
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Communication  Analysis  Example 
Face-To-Face  Homogeneous  vs  Asynchronous  Distributed  Heterogeneous 


%  Frequency  by  Process  States 


□  F2F  Homogeneous  □  AsynchDistr  Heter. 
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Communication  Analysis  Example 
Pathfinder  Results  (relationships  and  strength  of  relationships) 
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Pilot  Study  Results 
Time  Analysis  Example 
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Total  Task  Time  for  Face-to-Face  and  Asynchronous, 
Distributed  Hetergeneous  Teams 
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Pilot  Study  Results 
Time  Analysis  Example 
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Pilot  Study  Results 

Time  Analysis  Example 
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Face-to-Face  Homogeneous  vs.  Asynchronous  Distributed  Heterogeneous 

%  Time  in  Process  States 
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Pilot  Study  Results 

Collaboration  Maps  (Post  Session) 

Face-To-Face  Homogeneous  vs  Asynchronous, 
Distributed  Hetergeneous  Example 
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Questions: 

•Do  individual  mental  models  of  the 
collaboration  process  converge? 

•Do  individual  mental  models  correlate 
with  actual  team  performance? 


Convergence  of  Collaboration 
Stages  and  Knowledge  Types  by 
Team  Conditions 
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Problem  Definition 


Domain  Experience 


Recent  or  Planned  Publications  and 
Demonstrations 
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•  Warner,  N.W.,  Vanderwalker,  S.,  and  Verma,  N.  (Planned  Feb,  2003).  A  Conceptual  Model 

of  Knowledge  Building  during  Asynchronous,  Distributed  Collaboration.  Paper  for  journal 
article. 

•  Warner,  N.W.,  Vanderwalker,  S.,  and  Verma,  N.  (Planned  September,  2003).  The  Effect  of 
Collaboration  Mode  and  Knowledge  Distribution  on  the  Knowledge  Building  Process  in  a 
Asynchronous,  Distributed  Collaborative  Problem  Solving  Task.  Paper  for  journal  article. 

•  Warner,  N.W.,  NATO  RTO  Human  Factors  and  Medical  Panel  078/Task  Group  017: 
Uninhabited  Military  Vehicles  -  Augmenting  the  Force.  Invited  United  States  Navy 
Representative,  Leiden,  Netherlands,  June  2003. 
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Lessons  Learned 
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•  Achieved  better  collaboration  between  team  members  with  the  Survival  Scenarios 
compared  to  Age  of  Empires 

•  Concept  Maps  (Eden,  1992) 

•  Could  not  capture  individual  or  group  mental  models  of  the  knowledge  building  process 
during  asynchronous,  distributed  collaborative  problem  solving 

•  Concept  maps  are  good  at  capturing  and  representing  discrete  information  on  a  topic  but 
humans  have  difficulty  in  representing  abstract  information  with  this  open  ended 
technique 

•  Collaboration  Maps  (Warner,  2002) 

•  Computer  based  structured  approach  to  capturing  individual  mental  models  of 
collaboration  stages  and  types  of  knowledge  including  transformation  processes. 
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